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Introduction. The problem of automatic license plate recognition is considered. Its solution has many potential applications 
from safety to traffic control. The work objective was to develop an intelligent recognition system based on the application 
of deep learning algorithms, such as convolution neural networks that consider automotive standards for license plates in 
various countries and continents, and are tolerant to camera locations and quality of input images, as well as to changing 
lighting, weather conditions, and license plate deformations. 

Materials and Methods. An integrated approach for the problem solution based on the application of convolution neural 
network composition is proposed. An experimental analysis of neural network models trained to meet the requirements of 
the universal license plate recognition task was conducted. Based on it, models that showed the best ratio of quality and 
speed were selected. Quality of the system is provided through the optimization of various models with different 
modifications. In particular, convolution neural networks were trained using images from several datasets. In addition, to 
obtain the best results, the models used were pre-trained on a specially generated synthetic dataset. 

Results. The paper presents numerical experiments, the results of which imply the superiority of the developed algorithm 
over the commercial OpenALPR package on public datasets. In particular, on the 2017-IWT4S-HDR_LP-dataset, license 
plate recognition accuracy was 94 percent, and on the Application-Oriented License Plate dataset, 86 percent. 

Discussion and Conclusions. The resulting algorithm can be used to automatically detect and recognize license plates. The 
experiments show that the algorithm quality meets or exceeds quality of the commercial OpenALPR package. The 
developed algorithm quality can be improved through increasing the training dataset, which does not require the 
participation of the developer. 


Keywords: object detection and recognition, convolution neural networks, data generation and augmentation, license plate 
recognition 


For citation: A.V. Poltavski, T.G. Yurushkina, M.V. Yurushkin. Automatic license-plate recognition. Vestnik of DSTU, 
2020, vol. 20, no. 1, pp. 93-99. https://do1.org/10.23947/1992-5980-2020-20-1-93-99 





Introduction. Automatic License Plate Recognition (ALPR) systems are used for automatic speed control, 
identification of stolen vehicles, access control of vehicles in private premises and toll collection! [1,2,3]. However, most 
existing algorithms” ° work only for a specific license plate template or with complex image capture systems and are 
demanding on lighting conditions and types of vehicles [4,5,6]. 


Due to the rapid development of deep learning and its applications in the field of computer vision [7], it became 


possible to create an ALPR system capable of recognizing numerous license plate patterns in an arbitrary environment” 


' Lotufo RA, Morgan AD, Johnson AS. Automatic number-plate recognition. IEEE Colloquium on Image Analysis for Transport Applications. Feb, 1990. 
P. 1-6. 

* Du S, et al. Automatic license plate recognition (ALPR): A state-of-the-art review. IEEE Transactions on Circuits and Systems for Video Technology. 
2013;23(2):311-325. 

> Gou C., et al. Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines. IEEE Transactions on Intelligent 
Transportation Systems. 2016;17(4):1096—1107. 

* Hsu G., Chen J., Chung Y. Application-oriented license plate recognition. IEEE Trans. Veh. Technol. 2013;62(2):552—561. 

> Li H., Shen C. Reading car license plates using deep convolutional neural networks and LSTMs. 2016. arXiv preprint arXiv:1601.05610. 
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[8]. The fundamental obstacles in recognizing license plates are specific shooting conditions — the presence of rain, snow 
or poor lighting. The recognition task becomes more difficult if the license plate has an uncommon area and aspect ratio, 
background color, shape, number of lines, font size, distance between characters, etc. 

The study objective was to develop a license plate recognition system that supports various regional license plate 
standards, and does not depend on the conditions of video recording of cars, such as dirt on the numbers, weather 
conditions, etc. 

Materials and Methods 

Automatic system of detection and recognition of license plates. In the framework of this section, we give a 
description of the software implementation of the system of detecting a vehicle in the photograph, identifying the license 
plate in it, and its recognition. 

The proposed system consists of a composition of several neural network models. The idea of the composition is 
that the result of the previous model is fed to the input of the next one, and the whole analysis process is divided into stages. 


Schematically, the principle of the system is shown in Fig. 1: 


Vehicle detection Licensee plane detection License plate recognition 





Fig. 1. Three-step approach for license plate detection and recognition 


The first stage of the algorithm is to detect a vehicle. To solve this problem, we used the SSD Resnet” model, 
trained on a COCO (Common Objects in Context) * dataset. The result of this model is fragments of the input image which 
include vehicles. At the second stage, the obtained fragments are fed to the input of the developed model for detecting car 
numbers, which returns image fragments that include only car numbers. At the final stage, the characters that make up the 
car number are recognized and glued together. A detailed description of the developed models used in the second and third 
stages of the algorithm is given in the respective sections of the paper. 

The selected approach has the following advantages: 

e objects that can be perceived as a license plate are cut off: signs in showcases, windows or fences; 

e it becomes possible to establish a relationship between the number and the corresponding vehicle providing for 
the system expansion in the direction of recognition of other characteristics, such as the type, make and model of the 
vehicle, color and direction of movement, tracking in the video stream; 

e system modularity consisting of independent models specializing in solving a specific problem at a high level is 


maintained. 


License plate detection model. The initial data set for training the model was 1700 images of Indian cars taken at 


a certain angle. The model trained on this set was not variable for different conditions; therefore, the pseudo-labeling* 


‘Yuan Y., et al. A robust and efficient approach to license plate detection. IEEE Transactions on Image Processing. 2017;26(3):1102—1114. 

*Tensorflow detection model ZOO. Available at: http://download.tensorflow.org/models/object_ 
detection/ssd_resnet50_vl_fpn shared box predictor 640x640 cocol4 sync 2018 07 03.tar.gz (accessed: 03.11.2019). 

* COCO — Common Objects in Context. Available at: http://cocodataset.org/#home (accessed: 03.11.2019). 

* Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning / E. Arazo [et al.]. 2019. 8 Aug. // arXiv preprint arXiv:1908.02983. 
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method was applied for unlabeled data sets (images with vehicles). Thus, the initial data set for training was increased to 
400 thousand images. The size of the validation dataset is 5% of the training dataset. Images were randomly selected and 
checked for correct marking of license plate coordinates. 

To assess the quality of the model, IoU (Intersection over Untion), mAP (mean Average Precision) and AR 
(Average Recall) ' metrics were used: 

'DetectionBoxes_ Precision/mAP": class accuracy averaged over IoU threshold values in the range from 0.5 to 0.95 
in 0.05 increments; 

'DetectionBoxes_ Precision /mAP @.50IOU': average accuracy of classes by IoU value equal to 0.5; 

'DetectionBoxes_Precision/mAP (small)': average accuracy of classes for small objects (area <32%2 pixels); 

'DetectionBoxes Precision/mAP (medium)': average accuracy of classes for objects (322 pixels < area <96%2 
pixels); 

'DetectionBoxes_ Precision/mAP (large)': average class accuracy for large objects (96%2 pixels < area < 100002 
pixels). 

Table 1 


mAP result on validation data 


mAP 
mAP@.S010U] mAP@.7510U fmaP (smatt] "AP |onAP dares 





mee. ee ee Nel eages.|| pes 0.9739 0.7199 | 0.8197 | 0.8544 
detection vl FPN 


The resulting model is resistant to the size of the input image (Table 1), detects license plates at an angle, as well as 
with an uncommon aspect ratio. 

Vehicle plate numbers recognition model. The problem of license plate recognition is reduced to the task of 
detecting objects of 36 classes: 26 letters of the uppercase Latin alphabet and 10 digits. 

The initial data set for training the model was 1700 images of Indian cars. Validation took place at 10% of the 
initial set consisting mainly of hard-to-read images marked up manually. 

For lack of a sufficient number of labeled data sets that meet the requirements of training the numbers recognition 
model, it was decided to generate a synthetic data set. Such transformations as rotation, blurring and darkening, as well as 


various sizes and colorings of the license plate, font styles, and the distance between characters were used as augmentations 
(Fig. 2). 
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Fig. 2. Samples from generated datasets 


' mAP (mean Average Precision) for Object Detection. Available at: https://medium.com/@jonathan_hui/map-mean-average-precision-for-object- 
detection-45c121a31173 (accessed: 30.10.2019). 
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The model was trained on a synthetic dataset including 300 thousand images. To increase the accuracy and amount 
of real data for training the model, the pseudo-labeling technique was used, which increased the set of real data to 250 
thousand. Recognition accuracy on the validation dataset has increased by 15% reaching the mark of 63% (Fig. 4). This 


growth is due to the presence in real images of artifacts that are not available under generation. 


DetectionBoxes_Precision 


mAP 
tag: DetectionBoxes_Precision/mAP 





Fig. 3. mAP results of the model before and after adding real datasets 

















Table 2 
Comparison of mAP models on validation data 
mAP 
Task Model mAP | mAP@,S50IOU |mAP@,75IOU | mAP (small) mAP (large) 
(medium) 
Faster R-CNN | 9.4783 0.9372 0.3578 0.4689 0.5251 0.5543 
Resnet-101 
License 0.6332 0.9503 0.7228 0.6009 0.6397 0.7051 
plate 
ve 0.5567 0.9534 0.6113 0.5569 0.6094 0.6493 
Been Faster R-CNN 
Inception v2 
0.5784 0.9652 0.6184 0.5583 0.6157 0.6676 
































Research Results 
In the resulting combination of neural network models, each of the classifiers was independently trained using 


images from several datasets including license plates of countries from all continents and methods of increasing these sets to 
achieve stability under various conditions”. 

IoU metrics were used to evaluate license plate detection results. For the task of recognizing license plate 
characters, the image in which all the characters in the license plate were correctly recognized was considered correctly 
marked out. An odd, missing or incorrectly recognized symbol was considered an error on the whole license plate. 

The results were compared to the latest cloud version of the commercial OpenALPR’ package and the 2016 study* 


mentioned above. The testing was performed on the 2017-I[WT4S-HDR_LP-dataset (Table 3) provided in the study” and the 


' Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning. 

* Nowruzi FE, et al. How much real data do we actually need: Analyzing object detection performance using synthetic and real data. 2019. arXiv preprint 
arXiv:1907.07061. 

> OpenALPR Cloud APL. Available at: https://api.openalpr.com/v2/ (accessed: 02.11.2019). 

* Li H., Shen C. Op. cit. 

> Hsu G., Chen J., Chung Y. Op. cit. 
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Application-Oriented License Plate (AOLP) (Table 4) provided in the study’, which includes 2049 images with Taiwanese 
license plates. They were divided into three subsets with different levels of complexity and shooting conditions: Access 


Control (AC), Law Enforcement (LE) and Road Patrol (RP). 
























































Table 3 
Numerical experiment on public 2017-I[WT4S-HDR_LP-dataset 
Metrics Ours OpenALPR 
Number of correct license plate recognitions 619/653 377 / 653 
Percent of correct license plate recognitions mete STE, 
Table 4 
Numerical experiment on public dataset 
Metrics Applied solutions AC Subset (%) LE Subset (“%) RP Subset (%) 

ours 95.58 93.97 94.29 
OpenALPR 91.80 86.89 90.84 
[12] 1st approach(with CNN I) 93.53 89.83 86.58 
[12] 1st approach(with CNN II) 93.25 90.62 86.74 
IoU plate detection | [12] Ist approach(with CNN I 93.97 97.87 97-73 

& II) 

h(with global 
[12] 2nd approach(with globa 90.50 91.15 93.98 
features only) 
12]2 h(with both 
Eat eee eet 94.85 94.19 88.38 
local and global features) 
ours 88.75 83.94 85.40 
Number of correctly 
recognized license 
plates: 

OpenALPR 86.04 77.98 85.71 























' Hsu G., Chen J., Chung Y. Op. cit. 
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Discussion and Conclusions. Models were built to detect and recognize vehicle license plates. According to the 
numerical experiments, the model for detecting numbers turned out to be more accurate than the solutions obtained in 2016 
and 2019! * on the dataset published in 2013°. The license plate recognition model turned out to be more accurate than the 
decision of 2019* on the dataset obtained in 2017 [8]. To increase the accuracy of the models, methods of generating 
synthetic data and pseudo-labeling were used. The resulting system has appeared resistant to the size of the input image and 


the environmental properties. 
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